Vidiam: Corpus-based Development of a Dialogue Manager for Multimodal Question Answering

In this chapter we describe the Vidiam project, which concerns the development of a dialogue management system for multi-modal question answering dialogues as it was carried out in the IMIX project. The approach that was followed is data-driven, that is, corpus-based. Since research in Question Answering Dialog for multi-modal information retrieval is still new, no suitable corpora were available to base a system on. We report on the collection and analysis of three QA dialogue corpora, involving textual followup utterances, multimodal follow-up questions, and speech dialogues. Based on the data, we created a dialogue act typology which helps translate user utterances to practical interactive QA strategies. We then report how we built and evaluated the dialogue manager and its components: dialogue act recognition, interactive QA strategy handling, reference resolution, and multimodal fusion, using off-line analyses of the corpus data.